Networks for packet monitoring and replay

ABSTRACT

Disclosed herein are a system, non-transitory computer readable medium, and method for monitoring and replaying packets. A network tap forwards packets from a first network to a second network. At least one node in the first network has the same IP address as a node in the second network. The packets are replayed in the second network.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication No. 62/530,706 filed Jul. 10, 2017 the disclosure of whichis entirely incorporated herein by reference.

BACKGROUND

Computer networks heretofore may include a mesh of interconnectedservers, hubs, routers, switches, and storage arrays carrying criticalinformation. Such networks may be prone to infrastructure failures dueto network hardware changes and network congestion.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative computer apparatus in accordance with aspectsof the disclosure.

FIG. 2 is a flow diagram of an example method in accordance with aspectsof the disclosure.

FIG. 3 is an example network topology in accordance with aspects of thepresent disclosure.

FIG. 3A is an example network tap in accordance with aspects of thepresent disclosure.

FIG. 3B is an example packet in accordance with aspects of the presentdisclosure.

FIG. 3C is another example network in accordance with aspects of thepresent disclosure.

FIG. 4 is a further flow diagram of another example method in accordancewith aspects of the disclosure.

DETAILED DESCRIPTION

As noted above, networks may be prone to failures. In some networks, thesequence in which data packets are transmitted may be critical. Forexample, in video streaming networks, packets must typically arrive atthe playback device in the correct sequence so that the video showscorrectly. In a trading system, the correct sequence of the packets maybe important so that the correct state of the order is reflected on atrader's workstation. Furthermore, packets may be lost duringtransmission. In this instance, a network administrator may attempt torecover the lost packets. However, an administrator troubleshooting lostor out-of-sequence data packets may disrupt the performance of a liveproduction network. The administrator may need to execute troubleshooting software that may slow down a live data network being used bycustomers. Such a disruption may result in customer dissatisfaction,which in turn may lead to a loss of revenue.

In view of the foregoing, disclosed herein are an apparatus, method, andnon-transitory computer readable medium that monitors live productiondata packets and permits playback of these packets without disrupting alive production network. In one example, an apparatus may comprise anetwork interface and at least one processor to carry out the followingoperations: establish communication with a network terminal access point(TAP) of a first network, each node of the first network having aninternet protocol (IP) address; establish communication with at leastone node of a second network, each node in the second networkcorresponding to a node in the first network such that respective IPaddresses of corresponding nodes are equal; receive a first packet and asecond packet from the network TAP of the first network, each packetcomprising a source IP address and a timestamp, the source IP addressindicating a respective node in the first network from where each packetoriginates; and launch the first packet and the second packet from arespective node in the second network that corresponds to the source IPaddress of each packet in a sequence that is in accordance with thetimestamp of each packet.

In yet another aspect, a network TAP apparatus may comprise a pluralityof network interfaces and at least one processor. The at least oneprocessor may receive, via a network interface, a first packet from asource device. The first packet may be bound for a destination device.The source device and the destination device may be nodes of a firstnetwork. The source device and the destination device may each beassociated with a respective IP address. The at least one processor mayalso generate a duplicate packet that is a copy of the first packet. Thenetwork TAP may also forward, using another network interface, theduplicate packet to another destination device whose IP address isidentical to that of the destination device in the first network. Theother destination device may be a node in a second network differentfrom the first network.

The aspects, features, and advantages of the present disclosure will beappreciated when considered with reference to the following descriptionof examples and accompanying figures. The following description does notlimit the application; rather, the scope of the disclosure is defined bythe appended claims and equivalents.

FIG. 1 shows a schematic diagram of an illustrative computer apparatus100 for executing some of the techniques disclosed herein. Computerapparatus 100 may comprise a device capable of processing instructionsand transmitting data to and from other computers, including a laptop, afull-sized personal computer, a high-end server, or a network computerlacking local storage capability. Computer apparatus 100 may include allthe components normally used in connection with a computer. For example,it may have a keyboard and mouse and/or various other types of inputdevices such as pen-inputs, joysticks, buttons, touch screens, etc., aswell as a display, which could include, for instance, a CRT, LCD, plasmascreen monitor, TV, projector, etc. Computer apparatus 100 may alsocomprise a network interface 106 to communicate with other devices overa network. As will be noted further below, computer apparatus 100 may beused to store and replay packets,

The computer apparatus 100 may also contain at least one processor 102,such as processors from Intel® Corporation. In another example,processor 102 may be an application specific integrated circuit(“ASIC”). Memory 104 may store instructions that processor 102 mayretrieve and execute. In one example, memory 104 may be used by or inconnection with an instruction execution system that permits processor102 to fetch or obtain the logic from memory 104 and execute theinstructions contained therein.

Memory 104 may be a non-transitory computer readable medium (“CRM”),which may comprise any one of many physical media such as, for example,electronic, magnetic, optical, electromagnetic, or semiconductor media.Some examples of suitable non-transitory computer readable mediuminclude, but are not limited to, a portable magnetic computer diskettesuch as floppy diskettes or hard drives, a read-only memory (“ROM”), anerasable programmable read-only memory, a portable compact disc or otherstorage devices that may be coupled to computer apparatus 100 directlyor indirectly. The non-transitory CRM may also include any combinationof one or more of the foregoing and/or other devices as well.

As noted above, computer instructions stored in memory 104 may causeprocessor 102 to carry out one or more of the techniques disclosedherein. These instructions may comprise any set of instructions to beexecuted directly (such as machine code) or indirectly (such as scripts)by processor 102. In this regard, the terms “instructions,” “scripts,”or “modules” may be used interchangeably herein. The computer executableinstructions may be stored in any computer language or format, such asin object code or modules of source code. Furthermore, it is understoodthat the instructions may be implemented in the form of hardware,software, or a combination of hardware and software and that theexamples herein are merely illustrative.

As will also be discussed further below, computer apparatus 100 maystore and sort data packets for replay in database 108. These packetsmay be retrieved later for replay. Database 108 is not limited to anyparticular data structure. The data stored in database 108 may beformatted in any computer-readable format. Database 108 may comprisecomputer registers, a relational database with multiple tables arrangedwith fields and records, XML documents, or flat files. The stored datamay comprise any information sufficient to identify the relevantinformation, such as numbers, descriptive text, proprietary codes,references to data stored in other areas of the same memory or differentmemories (including other network locations) or information that is usedby a function to calculate the relevant data.

While FIG. 1 only depicts one processor, one memory, and one database,it is understood that computer apparatus 100 may actually compriseadditional processors, memories, and databases working in tandem thatmay or may not be stored within the same physical housing or location.To wit, although all the components of computer apparatus 100 arefunctionally illustrated as being within the same block, it will beunderstood that the components may or may not be stored within the samephysical housing.

One working example of the system, method, and non-transitory computerreadable medium is shown in FIGS. 2-3C. In particular, FIG. 2illustrates a flow diagram of an example method 200 for monitoring andreplaying packets. FIGS. 3-3C show a working example in accordance withthe techniques disclosed herein. The actions shown in FIGS. 3-3C will bediscussed below with regard to the flow diagram in FIG. 2.

Referring to FIG. 2, a processor (e.g., a processor 102 of computerapparatus 100) may establish communication with a network TAP of a firstnetwork, as shown in block 202. In block 204, a processor may alsoestablish communication with at least one node of a second network.Referring now to FIG. 3, an example network topology in accordance withaspects of the disclosure is shown. FIG. 3 illustrates a network 301 anda network 301P. In the example of FIG. 3, network 301 may be aproduction environment that includes workstations 302 and 314; switches306 and 310; and, network TAPS 304 and 308. Network 301P may be a packetmonitoring environment that includes workstations 302P and 314P;switches 306P and 310P; hub 316; computer apparatus 100; and database108. In the example of FIG. 3, at least one node in the second network301P may correspond to a node in the first network 301 such thatrespective IP addresses of corresponding nodes are equal. For example,workstation 302 and 302P may have identical IP addresses. Furthermore,switches 306 and 306P and switches 310 and 310P may also have identicalIP addresses. The advantage of having these identical IP addresses willbe explained further below.

The workstations 302, 314, 302P, and 314P may also have at least oneprocessor, at least one memory, and at least one network interface likecomputer apparatus 100 shown in FIG. 1. Networks 301 and 301P may belocal area networks (“LAN”) or a wide area networks (“WAN”). A LAN mayinclude, for example, an Ethernet 10/100 LAN or a gigabit Ethernet LAN.Networks 301 and 301P may be connected to a service provider via a cablenetwork, a digital subscriber line (DSL) network, a T1 or T3 network, amicrowave network, a WiMax (IEEE 802.16) network, or the like.Furthermore, networks 301, 301P, and the intervening nodes therein mayuse various protocols including virtual private networks, local Ethernetnetworks, and private networks using communication protocols proprietaryto one or more companies, cellular and wireless networks, HTTP, andvarious combinations of the foregoing. In one example, networks 301 and301P may be wireless networks that conform to standards includingBluetooth®, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.16, orthe like. It is understood that the network topologies shown in FIG. 3are merely illustrative and that many different topologies may beimplemented. Furthermore, it is understood that a network topology mayinclude many more workstations, hubs, switches, servers, and networkTAPS and that the example of FIG. 3 shows a small number of nodes forease of illustration only.

Network 301 is shown with a workstation 302 in communication withanother workstation 314. In between workstation 302 and workstation 314,there are two network switches, switch 306 and switch 310, and threenetwork TAPS 304, 308, and 312. As noted above, at least one device ofnetwork 301 may have a corresponding device in network 301P. Thesecorresponding devices may have identical IP addresses. That is, if theIP address of switch 306 is 155.22.76, 222, the IP address of thecorresponding switch 306P may be the same. By way of example,workstations 302 and 314 of network 301 correspond to workstations 302Pand 314P of network 301P respectively. Switches 306 and 310 of network301 correspond to switches 306P and 310P in network 301P respectively.

Each switch in the network shown in FIG. 3 may comprise a memory, and aprocessor. Further, each switch may include a number of data ports, suchas uplink data ports and downlink data ports. One or more switches inthe networks of FIG. 3 may also include data comprising flow tables. Forexample, each entry in a given flow table may include a key and anaction. As a switch receives packets, header information in thosepackets may be matched against the keys of the flow table to determine acorresponding action such as a next hop. The entries in the flow tablemay be used directly or indirectly to forward packets. While each switchin FIG. 3 is depicted as hardware switches, a software based switch maybe used in other examples. In this instance, the flow table may beaccessed directly by a forwarding software module to alter the packet'sheader information and to forward the packet to an appropriate port.Alternatively, a processor of a switch may program hardware modulesbased on the flow table entries, and these hardware modules may forwardpackets based on each flow's match criteria, action, and priority. Asnoted above, network 301 may be a live or primary network where usersare transferring real-time data between workstation 302 and workstation314. In contrast, network 301P may be used for capturing the packetsand/or replaying the packets.

Each network TAP 304, 308, and 312 of network 301 may comprise hardwarethat duplicates each packet flowing between a respective pair of networknodes (i.e., network TAP 304 mirrors bi-directional packets flowingbetween workstation 302 and switch 306, network TAP 308 mirrorsbi-directional packets flowing between switch 306 and switch 310, and soon). The duplicated packet may be forwarded to the device in network301P that corresponds to the destination device in network 301. By wayof example, workstation 302 may transmit a packet destined for switch306. In this instance, network TAP 304 may create a duplicate of thepacket and forward that duplicate to the corresponding switch 306P innetwork 301P. By way of further example, if a packet is traveling fromswitch 306 to workstation 302, network TAP 304 may create a duplicateand forward that duplicate to workstation 302P. As such, each networkTAP may forward duplicates to the corresponding destination node innetwork 301P depending on the direction of the packet.

As noted earlier, corresponding devices in networks 301 and 301P mayhave identical IP addresses. By using an identical IP address, theduplicate packet created by a network TAP would automatically route tothe corresponding device in network 301P without needing additionallogic in the network TAP to alter the destination address. For example,if a network TAP in the production environment forwarded all theduplicate packets to a monitoring device with a unique IP address, thedestination IP address of each duplicate packet may need to be changedso that each packet routes accordingly. Changing the destination IP ofeach packet may be a burden on the production environment and may causefurther delays.

The switches and workstations of network 301 and their counterparts innetwork 301P may all be time synchronized. By way of example, if networkTAP 304 of FIG. 3 copies a packet and forwards the duplicate to switch306P and the timestamp of the duplicate is not synchronized with theinternal clock of switch 306P, switch 306P may reject the duplicatepacket. Therefore, in one example, a timeserver (not shown) may belinked to the switches and workstations of network 301 and theircounterparts in network 301P. The time server may include, for example,a GPS satellite antenna wired to a grandmaster precision time protocol(PTP) clock. Thus, the PTP protocol may be used to synchronize clocksthroughout networks 301 and 301P. The PTP may be in accordance to thestandards specified in IEEE 1588-2002 and IEEE 1588-2008.

The network TAPS shown in FIG. 3 may be designed to mirror the trafficwithout impeding the flow of the production traffic flowing in network301. Referring now to FIG. 3A, a detailed illustration of a network TAP600 in accordance with the present disclosure is shown. Each network TAPmay also comprise a processor 602 and memory 608. The first networkinterface 604 may be coupled to wired or wireless networks. In oneexample, the first network interface 604 may comprise a plurality ofports configured to permit bi-directional traffic to pass throughnetwork TAP 600. The second network interface 606 may provide access toa device in second network 301P. That is, the duplicate packets may beforwarded via the second network interface 606. However, in otherexamples, first network interface 604 may permit the bi-directionaltraffic as well as forward the duplicate packets to network 301P. Memory608 of network TAP 600 may include network access instructions.

Each network TAP may be a switched port analyzer (SPAN) or remote switchport analyzer (RSPAN) TAP that makes copies of each packet passingbetween devices in the network. In one example, each network TAP shownin FIG. 3 may be an optical fiber TAP. An optical fiber TAP may providethe exact duplicate of the signal on the network link without anydisruption to the network activity. Optical fiber TAPS may continuallypass data on its ports, without either modifying or degrading the signalpassing through. The Optical fiber TAP may provide a duplicate of eachpacket passing by splitting a small amount of light flowing on thetapped network link. In on example, the network TAPS shown in thefigures may be active Optical fiber TAPS, which use electricity foroperation, or passive Optical fiber TAPS that do not use electricity.

Referring back to the example of FIG. 3, packets received bycorresponding devices in network 301P (i.e., workstations 302P/314P andswitches 306P/310P), may be forwarded to hub 316. Hub 316 may be aseries of packet handling switches that route all the packets tocomputer apparatus 100. Hub 316 may insert other relevant information inthe packet. By way of example, if network 301 is used for real-timetrading of financial instruments, Hub 316 may ensure that all therelevant identifiers are included in the packet (e.g., order identifier,trade identifier, etc.).

All the packets received by computer apparatus 100 from Hub 316 may bestored in database 108 and the packets may be sorted by timestamp. Thatis, computer apparatus 100 may store all the packets transferred betweenthe devices of network 301 in database 108 by way of network 301P. Asnoted above, the users of network 301 may be traders and the packets mayrepresent orders for financial instruments or execution of trades forfinancial instruments. In this instance, a second packet may beassociated with the first packet by way of an order identifier,execution identifier, etc. That is, the second packet may have anidentifier that is identical to or related in some way with the firstpacket (e.g., each packet may be a different transaction on the sameorder). Referring now to FIG. 3B, an example packet representing a tradefor a financial instrument is shown. The illustrative packet of FIG. 3Bmay comprise transport protocol details 402, a source internet protocol(“IP”) address 404, and a user identifier field 406. The illustrativepacket may also comprise a financial instrument field 408 that maycontain a symbol of a stock or bond, and a price field 410 that mayrepresent a bid price, ask price, or execution price. The illustrativepacket may also contain a size field 412 that may represent an amount ofthe instrument being bought, sold, or otherwise executed, and atimestamp field 414 that may represent the time in which a particularnetwork node generated or forwarded the packet. The illustrative packetmay also include a destination IP address field 416. The precision oftimestamp field 414 may be set to nanoseconds, however it is understoodthat different levels of precision may be used. As noted above, in oneexample, the timestamps between networks 301 and 301P are synchronized.

Referring back to FIG. 2, computer apparatus 100 may launch packets inthe second network from a respective node in the second network thatcorresponds to the source IP address in each packet, as shown in block208. In one example, the monitoring network 301P shown in FIG. 3 may beused to replay the packets. However, a separate replay network may alsobe used. FIG. 3C illustrates a working example of a separate replaynetwork. The network in FIG. 3C may be used for replay and analysisinstead of the networks shown in FIG. 3 to further avoid any risk ofdisrupting the production environment of network 301. A separate replaynetwork may also be advantageous if disruption to the monitoring network301P is also necessary. In this instance, network 301R shown in FIG. 3Cmay be used for replay in lieu of network 301P. However, it isunderstood that network 301P may still be used for replay and analysis.Network 301R of FIG. 3C may have workstations 302R and 314R and switches306R and 310R. The workstations and switches shown in network 301R mayalso have identical IP addresses as their respective correspondingdevices in network 301. That is, the IP addresses of workstations 302Rand 314R may be identical to the IP addresses of workstations 302 and314 in network 301 of FIG. 3 respectively. Similarly, the IP addressesof switches 306R and 310R may have the identical IP addresses asswitches 306 and 310 in network 301 respectively. While FIG. 3C showscomputer apparatus 100 also used for replay, it is understood that adifferent computer apparatus may be used for replay. Each packet indatabase 108 may include a source IP address, an identifier, and atimestamp. In addition, the plurality of packets may be sorted in thedatabase by timestamp and identifier. Also, in other implementations,the IP addresses of the replay devices in network 301R may be differentfrom their counterparts in the production network. In this instance, thesource and destination IP addresses of each packet may need to bealtered before replay. This change of IP addresses may not disturb theproduction and mirror networks shown in FIG. 3.

As noted above, each packet in database 108 may comprise a source IPaddress, an identifier, and a timestamp. Referring back to the workingexample of FIG. 3, a packet travelling from workstation 302 toworkstation 314 in network 301, may have a total of three copies storedin database 108. By way of example, a first copy may be generated bynetwork TAP 304, a second copy may be generated by network TAP 308, anda third copy may be generated by network TAP 312. Thus, a snapshot ofthe packet as it travels through the network may be captured in database108. By way of further example, the first, second, and third copies maybe associated with a particular order of a financial trade.

Referring back to FIG. 3C, computer apparatus 100 may launch theplurality of packets from the corresponding source IP address in thesecond network (e.g., network 301R of FIG. 3B) in a sequence that is inaccordance with the timestamp of each packet. This allows computerapparatus 100 to reproduce an initial route of each packet as it shouldhave been in the first network (e.g., network 301). As noted above, atleast one device in the network 301 has a corresponding device innetwork 301R.

As also noted above, three copies of a packet traveling from workstation302 to workstation 314 in FIG. 3 may be stored in database 108. NetworkTAPS 304, 308, and 312 may generate each copy respectively. The firstcopy of the packet may have an earlier timestamp as the second packet,and the second packet may have an earlier timestamp than the thirdpacket. The three packets may be sorted such that the packet with theearliest timestamp may be launched first, the packet with the secondearliest timestamp may be launched second, and so on. In a tradingsystem scenario, the packets may also be sorted by order identifier suchthat the packets of each order are grouped together in the database.

Referring back to FIG. 3C, computer apparatus 100 may retrieve thepackets for a particular order, such as a first-in-first-out order basedon the timestamp. In the event an administrator desires to launch apacket from workstation 302, computer apparatus 100 may transmit thefirst copy to workstation 302R to permit workstation 302R to launch thepacket to workstation 314R again. This allows the system to replay apacket from different points in the network to determine where thepacket was lost or where the packet encountered network congestion. Inthe event an administrator would like to play the second packet producedby network TAP 308 as the packet travelled from switch 306 to switch310, computer apparatus 100 may retrieve and transmit the second packetto switch 306R and allow the packet to travel from switch 306R toworkstation 314. Since these packets already include a destination IPaddress, the packets would automatically route to the destination node.

Referring now to FIG. 4, an example method 500 that may be executed by anetwork TAP in network 301 is shown. In block 502, a network TAP mayreceive, via a network interface, a first packet from a source device,the first packet being bound for a destination device. As noted above,the source device and the destination device may be nodes of a firstnetwork, such as switch 306 and switch 310 of network 301. The sourcedevice and the destination device may each be associated with arespective IP address. In block 504, a network TAP may generate aduplicate packet that is a copy of the first packet. The network TAP mayfurther permit packets to proceed toward the destination device in thefirst network. For example, in FIG. 3, network TAP 308 may permit apacket to flow between switch 306 and switch 310. In block 506, anetwork TAP may forward, using another network interface, the duplicatepacket to another destination device having an IP address identical tothat of the destination device in the first network. The otherdestination device may be a node in a second network different from thefirst network. For example, network TAP 308 in FIG. 3 may forwardduplicate packets to either switch 306P or 310P depending on thedirection in which the packet is traveling. As noted above, the switchesand workstations in network 301P of FIG. 3 may have identical IPaddresses as their counterparts in network 301 to reduce the burden onthe production network.

Advantageously, the above-described system, non-transitory computerreadable medium, and method permit monitoring of packets at differentpoints in a network by using alternate networks with devices having thesame IP address as some of the devices in the original network. Thisallows the TAPS of the live network to make exact copies of the packetswithout altering the destination IP address of each packet. Furthermore,this allows the packets to be analyzed and replayed without disruptingthe production environment

Although the disclosure herein has been described with reference toparticular examples, it is to be understood that these examples aremerely illustrative of the principles of the disclosure. It is thereforeto be understood that numerous modifications may be made to the examplesand that other arrangements may be devised without departing from thespirit and scope of the disclosure as defined by the appended claims.Furthermore, while particular processes are shown in a specific order inthe appended drawings, such processes are not limited to any particularorder unless such order is expressly set forth herein. Rather, varioussteps can be handled in a different order or simultaneously, and stepsmay be omitted or added. Furthermore, while some examples noted aboverefer to using the techniques herein in financial trading environments,it is understood that the techniques disclosed herein may be used in anytype of production network environment, such as movie streaming, musicstreaming, or the like.

The invention claimed is:
 1. An apparatus comprising: a networkinterface; at least one processor to: establish communication with anetwork terminal access point (TAP) of a first network, each node of thefirst network having an internet protocol (IP) address; establishcommunication with at least one node of a second network, at least onenode in the second network corresponding to a node in the first networksuch that respective IP addresses of corresponding nodes are equal;receive a first packet and a second packet from the network TAP of thefirst network, each packet comprising a source IP address and atimestamp, the source IP address indicating a respective node in thefirst network from where each packet originates; and launch the firstpacket and the second packet from a respective node in the secondnetwork that corresponds to the source IP address of each packet in asequence that is in accordance with the timestamp of each packet.
 2. Theapparatus of claim 1, wherein the at least one processor is furtherconfigured to store the first packet and the second packet in adatabase.
 3. The apparatus of claim 2, wherein the at least oneprocessor is further configured to sort the first packet and the secondpacket in the database by timestamp.
 4. The apparatus of claim 2,wherein the at least one processor is further configured to retrieve thefirst packet and the second packet from the database to launch the firstpacket and the second packet.
 5. The apparatus of claim 1, wherein thenetwork tap is an optical fiber network tap.
 6. The apparatus of claim1, wherein an internal clock of a node in the first network issynchronized with an internal clock of a node in the second network. 7.The apparatus of claim 6, wherein the node of the first network and thenode of the second network communicate with a time server in order tosynchronize the respective internal clocks.
 8. The apparatus of claim 1,wherein the first packet is associated with the second packet by way ofan identifier field in the first packet and the second packet.
 9. Amethod comprising: establishing, by at least one processor,communication with a network terminal access point (TAP) of a firstnetwork, each node of the first network having an internet protocol (IP)address; establishing, by at least one processor, communication with atleast one node of a second network, at least one node in the secondnetwork corresponding to a node in the first network such thatrespective IP addresses of corresponding nodes are equal; receiving, byat least one processor, a first packet and a second packet from thenetwork TAP of the first network, each packet comprising a source IPaddress and a timestamp, the source IP address indicating a respectivenode in the first network from where each packet originates; andlaunching, by at least one processor, the first packet and the secondpacket from a respective node in the second network that corresponds tothe source IP address of each packet in a sequence that is in accordancewith the timestamp of each packet.
 10. The method of claim 9, furthercomprising storing, by the at least one processor, the first packet andthe second packet in a database.
 11. The method of claim 10, furthercomprising sorting, by at least one processor, the first packet and thesecond packet in the database by timestamp.
 12. The method of claim 10,further comprising retrieving, by at least one processor, the firstpacket and the second packet from the database to launch the firstpacket and the second packet.
 13. The method of claim 9, wherein thenetwork tap is an optical fiber network tap.
 14. The method of claim 9,wherein an internal clock of a node in the first network is synchronizedwith an internal clock of a node in the second network.
 15. The methodof claim 9, wherein the first packet is associated with the secondpacket by way of an identifier field in the first packet and the secondpacket.